CSS-LM: A Contrastive Framework for Semi-Supervised Fine-Tuning of Pre-Trained Language Models

Fine-tuning pre-trained language models (PLMs) has demonstrated its effectiveness on various downstream NLP tasks recently. However, in many low-resource scenarios, the conventional fine-tuning strategies cannot sufficiently capture important semantic features for tasks. To address this issue, we introduce a novel framework (named "CSS-LM") to improve phase of PLMs via contrastive semi-supervised learning. Specifically, given specific task, retrieve positive and negative instances from large-scale unlabeled corpora according their domain-level class-level relatedness task. We then perform learning both retrieved original labeled help crucial task-related features. The experimental results show that CSS-LM achieves better than strategy series with few-shot settings, outperforms latest supervised strategies. Our datasets source code will be available provide more details.